{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a6643a3f",
   "metadata": {},
   "source": [
    "## Transferring Files To and From Azure Blob Storage\n",
    "\n",
    "Retrieve files from an Azure Blob Storage container before executing a task, then upload files to a storage container after the task's execution.\n",
    "\n",
    "The following example workflow downloads an image file from a storage container, processes the file's contents, then uploads the file back to the storage container.\n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "Upload a color image file to a storage container. Make note of the blob URI to use in the workflow, in the format `https://<storage_account_name>.blob.core.windows.net/<container_name>/<blob_name>`.\n",
    "\n",
    "In this example, the blob URI is `https://covalenthowto.blob.core.windows.net/howto/remote_{unprocessed_filename}`, where `{unprocessed_filename}` is a variable containing the name of the file.\n",
    "\n",
    "Additionally, create a service principal whose credentials will be used to authenticate to the storage account.\n",
    "\n",
    "### Procedure\n",
    "\n",
    "1. Define two Covalent `FileTransfer` objects and a Covalent `Blob` strategy object. In this example, we will be using factory classes `TransferFromRemote` and `TransferToRemote` which generate `FileTransfer` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1922dd68",
   "metadata": {},
   "outputs": [],
   "source": [
    "import covalent as ct\n",
    "from typing import List, Tuple\n",
    "from pathlib import Path\n",
    "from skimage import io, color\n",
    "\n",
    "strategy = ct.fs_strategies.Blob(\n",
    "    client_id=\"my-sp-client-id\",\n",
    "    client_secret=\"my-sp-client-secret\",\n",
    "    tenant_id=\"my-aad-tenant-id\",\n",
    ")\n",
    "\n",
    "unprocessed_filename = \"unprocessed_file.png\"\n",
    "processed_filename = \"processed_file.png\"\n",
    "\n",
    "unprocessed_filepath = str(Path(unprocessed_filename).resolve())\n",
    "processed_filepath = str(Path(processed_filename).resolve())\n",
    "\n",
    "storage_account = \"covalenthowto\"\n",
    "storage_container = \"howto\"\n",
    "\n",
    "blob_source_path = f\"https://{storage_account}.blob.core.windows.net/{storage_container}/remote_{unprocessed_filename}\"\n",
    "blob_dest_path = f\"https://{storage_account}.blob.core.windows.net/{storage_container}/remote_{processed_filename}\"\n",
    "\n",
    "ft_1 = ct.fs.TransferFromRemote(blob_source_path, unprocessed_filepath, strategy=strategy)\n",
    "ft_2 = ct.fs.TransferToRemote(blob_dest_path, processed_filepath, strategy=strategy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea1e74e6",
   "metadata": {},
   "source": [
    "2. Define an electron to:\n",
    "    1. Download the unprocessed file from Blob storage\n",
    "    2. Perform some processing on the contents\n",
    "    3. Upload the processed file to Blob storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c57760",
   "metadata": {},
   "source": [
    "Access the file paths inside the electron as shown below using the \"files\" keyword argument. Covalent injects the source and destination file paths of the `TransferFromRemote` and `TransferToRemote` objects into the `files` argument. In this case, the `files` variable is a list of tuples of the form `(<source-path>, <destination-path>)`. The list looks something like this:\n",
    "\n",
    "```python\n",
    "[('/remote_unprocessed_file.png', '/path/to/current/dir/unprocessed_file.png'), ('/path/to/current/dir/processed_file.png', '/remote_processed_file.png')]\n",
    "```\n",
    "\n",
    "The Blob storage account and container names are omitted from the remote path in the list; they are applied automatically by the `FileTransfer` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cf73da5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "@ct.electron(files=[ft_1, ft_2]) # ft_1 is done before the electron is executed; ft_2 is done after.\n",
    "def to_grayscale(files: List[Tuple[str]] = None):\n",
    "\n",
    "    # Get the downloaded file's path\n",
    "    image_path = files[0][1] # destination file path of first file transfer, downloaded before executing this electron\n",
    "    \n",
    "    # Convert the image to grayscale\n",
    "    img = io.imread(image_path)[:, :, :3] # limiting image to 3 channels\n",
    "    gray_img = color.rgb2gray(img)\n",
    "\n",
    "    # Save the grayscale image to the upload file path\n",
    "    gray_image_path = files[1][0] # source filepath of second file transfer, to be uploaded\n",
    "    io.imsave(gray_image_path, gray_img, mode=\"L\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2a8f68e",
   "metadata": {},
   "source": [
    "3. Create and dispatch a lattice to run the electron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5831c062",
   "metadata": {},
   "outputs": [],
   "source": [
    "@ct.lattice\n",
    "def process_blob_data():\n",
    "    return to_grayscale()\n",
    "\n",
    "dispatch_id = ct.dispatch(process_blob_data)()\n",
    "status = ct.get_result(dispatch_id, wait=True).status\n",
    "print(status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7600fd65",
   "metadata": {},
   "source": [
    "### See Also\n",
    "\n",
    "[Transferring Local Files During Workflows](./file_transfers_for_workflows_local.ipynb)\n",
    "\n",
    "[Transferring Files To and From a Remote Host](./file_transfers_to_from_remote.ipynb)\n",
    "\n",
    "[Transferring Files To and From an S3 Bucket](./file_transfers_to_from_s3.ipynb)\n",
    "\n",
    "[Transferring Files To and From Google Cloud Storage](./file_transfers_to_from_gcp_storage.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
